Tuning TA

Configuration and Tuning

This section will walk you through how to tune the application (either Master or Client Manager) to get better performance.

Note: Any component (master, client manager, etc) running on java 17 will not start if “-XX:PermSize=XXX -XX:MaxPermsize=XXX” parameters are present in the JVMARGS. They must be removed from all *.props files for that component to start.

Memory

These parameters tune the memory size the application should need to get work done. If an application has insufficient memory, and out-of-memory error is thrown due to poor performance of the application.

The parameters apply to all Java applications:

  • -Xmn = size of young generation (1/4 size of heap)

  • -Xmx = max heap size

  • -Xms = initial heap size (guarantees JVM has that much memory)

  • -Xss = thread stack size (increase if getting stack overflow exceptions)

  • JVMARGS= -Xmn1024m -Xms28672m -Xmx28672m

Java Garbage Collection

To make heap memory available to the Java application, the Java Virtual Machine (JVM) periodically runs garbage collection to recycle unused heap memory. When major Java garbage collection occurs, the system pauses application execution for a brief duration. This pause duration depends on the application and the efficiency of the Java garbage collection scheme. Configuring the right Java garbage collection scheme is important to ensure that the Java application consumes the optimum amount of memory and is not paused frequently or for longer durations.

Example: By default, Java 11, uses the G1 garbage collection scheme. The G1 garbage collection scheme aims at minimizing the duration of the application pause. By default, G1 aims at limiting the application pause duration to 200 milliseconds. Giving G1 a short duration to do major garbage collection might result in high heap memory consumption.

To avoid high heap memory consumption, increase the application pause duration by setting the JVM parameter:

-XX:MaxGCPauseMillis

If the Java application running on Java 11 still consumes high memory, set the JVM parameter:

XX:+UseParallelGC

Memory Settings for TA Adapters

The following parameter in the <Adapter GUID>\config\service.props file specifies the initial (16 MB) and maximum (256 MB) heap memory size of the JVM required for TA Adapters.

JVMARGS=–Xms16m –Xmx256m

While the memory setting above applies to all other Adapters, these Tidal Adapters require additional heap memory:

  • Tidal Automation Adapter for Azure Blob

  • Tidal Automation Adapter for Azure Virtual Machine

  • Tidal Automation Adapter for Salesforce

  • Tidal Automation Adapter for SAP Financial Closing Cockpit

Tidal Automation Adapter for SAP Solution Manager Set the JVM parameter for these Adapters to:

JVMARGS=–Xms256m –Xmx1024m

CPU

The system CPU is not readily tunable from our application or the JVM itself, however, know that the CPU is an important resource

that directly impacts performance. If a system does not have enough CPUs or CPU power, applications can run very slowly across the

board. Upgrade your system to more CPUs or faster CPUs if the system monitor consistently shows the CPU meter very high.

JMS

The the Master and Client Manager rely on the Java Message Service (JMS) for internal and cross communication with each other. You can think of JMS as the communication link between the Master and Client Manager. That is why JMS can greatly impact performance, especially if there is a lot of data moving back and forth between the Master and Client Manager.

JMS sessions process all messages such as during the use of one worker thread per session.

These threads are affected:

  • MinSessionPoolSize – minimum number of ActiveMQ sessions kept pooled. Having sessions available to handle requests reduces the cost of allocating sessions on demand.

  • MaxSessionPoolSize – maximum number of ActiveMQ sessions kept pooled.

    Note: If more sessions are needed to process messages, the system will still allocate them. This setting only limits the number of sessions allowed to be pooled.

  • MaxConcurrentMessage – maximum number of ActiveMQ messages allowed to be processed concurrently. This setting is important for increasing throughput and utilizing all the cores on a system when there are many messages waiting in the ActiveMQ queues.

These Message Brokers handle all JMS traffic:

  • MessageBroker.MemoryLimit – how much memory (in MB) to allocate to ActiveMQ for storing in-flight messages. If queues become full, ActiveMQ will page messages to disk, which is more expensive than keeping them in memory.

  • MessageBroker.TempLimit – how much memory (in MB) to allocate to ActiveMQ for storing temporary messages. If queues become full, ActiveMQ will page messages to disk, which is more expensive than keeping them in memory.

  • MessageBroker.StoreLimit – how much disk space (in MB) to allocate to ActiveMQ for storing in-flight messages when memory is full.

These master.props properties are what you would tune for DSP to Master message traffic for all DSP connections:

  • MinSessionPoolSize (5)

  • MaxSessionPoolSize (10)

  • MaxConcurrentMessages (10)

These master.props properties are what you would tune for DSP to Master message traffic per DSP connections:

  • ClientConnection.MinSessionPoolSize (2)

  • ClientConnection.MaxSessionPoolSize (5)

  • ClientConnection.MaxConcurrentMessages (5)

These master.props properties are what you would tune for Remote Master to Master message traffic per Remote Master:

  • RemoteMasterClient.MinSessionPoolSize (2)

  • RemoteMasterClient.MaxSessionPoolSize (5)

  • RemoteMasterClient.MaxConcurrentMessages (5)

These master.props properties are what you would tune for Master to Remote Master message traffic per Remote Master:

  • RemoteMasterServer.MinSessionPoolSize (2)

  • RemoteMasterServer.MaxSessionPoolSize (5)

  • RemoteMasterServer.MaxConcurrentMessages (5)

These tes-6.5.dsp properties are what you would tune for Master to DSP message traffic:

  • ClientNode.MinSessionPoolSize (5)

  • ClientNode.MaxSessionPoolSize (10)

  • ClientNode.MaxConcurrentMessages (10)

These tes-6.5.dsp properties are what you would tune for Fault Monitor to DSP message traffic:

  • FTNode.MinSessionPoolSize (2)

  • FTNode.MaxSessionPoolSize (5)

  • FTNode.MaxConcurrentMessages (5)

These master.props property is what you would tune for the connections used to read/write to the Master database:

  • DatabaseConnections (20)

Master Messaging

Master messaging has a big impact on the performance of the Master and thus, indirectly, the Client Manager as well. The Master is designed to be a heavily multi-threaded application. By tuning the messaging parameters, the Master is better able to utilize the threading capabilities of the system.

All work in the Master is performed by these message threads:

  • Message Queues

    Before a message is sent to an object, it is first posted to one of the application’s message queues. Each queue or set of queues is allocated for a specific type of message.

    Example: The default queues are for general messages, the compiler queue is for compile messages, and the communication queues are for communication messages. Each queue has a priority that determines how often its messages are processed. Messages in higher-priority queues are processed more frequently than messages in lower-priority queues.

  • Message Threads

    Message threads are responsible for pulling messages out of queues and then executing them. Depending on the application configuration, there may be anywhere from a dozen to a hundred threads running inside the application. Generally speaking, more threads equal better performance, since each CPU can execute a thread at the same time as another CPU. So, technically, a dual core system can process twice as many messages as a single core system. However, in reality, because each thread consumes some memory and CPU, performance is expected to degrade when there are too many threads for the system to handle.

  • Message Thread Pools

    A message thread pool groups together a set of messages threads with a set of message queues. The intent is to force threads from a pool to only process messages for queues from the same pool. This guarantees if threads in one pool are busy, messages in another pool will still get processed by free threads in that pool.

To configure the Master messaging:

  • MessageThreads = Pulls messages out of the queue and executes them.

  • SpecialMessageThreads = adapter-related threads – The threads are used in the Master for adapter service requests that come from the adapter’s UI and are passed on to adapters. These threads can also be used by adapters that are built into the master such as AS400.

  • EventMessageThreads = event-related threads – The threads are used in the Master for executing actions such as sending emails, sending SNMP traps, and so on.

  • CommThreads = Communications threads

These clientmgr.props properties can be tuned for the Client Manager to utilize the threading capabilities of the system:

  • SpecialMessageThreads – This is the thread pool that handles the data updates for objects.

  • EventMessageThreads – This is the thread pool that handles new cache objects for SystemActivityMessage and MessageLog messages.

  • ActivityMessageThreads – This optional Activity thread pool is used by the TESPlugin to process Job and Jobrun Messages.

    Note: As a best practice, retain no more than 20 million message log records for better performance.

Master DB tuning

To enhance the performance on the Oracle DB, execute these query on the Admiral DB, so that the cache can be set to sequence (msglog_seq1):

ALTER SEQUENCE customers_seq CYCLE CACHE <number of records to cache>;

Example: ALTER SEQUENCE customers_seq CYCLE CACHE 5

For more information on sequence statement, see Create Sequence.

Implementing In-Memory Cache Changes

When the Client Manager is started, the in-memory cache is created and initialized based on the data definition configuration file (tescache.xml). A configurable range of production days is loaded into the cache as specified by the CacheJACHistory and CacheJACLookAhead parameters. A configurable number of rows is loaded into the cache as specified by the CacheMsgLogNumRows parameter. This process is done on a separate thread from the Client Manager main startup sequence.

Viewing In-Memory Cache

From the menu, choose Reports > System Info > In Memory Cache. The In Memory Cache view appears as shown in the figure.

The dsp file properties are set to override the default values of the in-memory cache:

Name

Description

Default Value

Value

Mandatory

CacheJdbcReconnects

Number of times to retry connections to the Database. A reconnection attempt can be made in every 10 seconds by default. This is not a configurable parameter.

2

Numeric

No

CacheJdbcConnections

Number of connections to the Database.

20

Numeric

No

CacheJACLookAhead

Number of production days in future to cache Job Activity data.

1

Numeric

No

CacheJACHistory

Number of production days in the past to cache Job Activity data.

1

Numeric

No

CacheMsgLogNumRows

Number of message log rows to be cached (only the most recent rows are cached and older data is purged at every groom interval).

10000

Numeric

No

CacheGroomInterval

Interval in milliseconds to groom the Cache, items groomed in the cache can typically be one of these types:

  • Purge older rows from message logs.

  • A production day once loaded into cache is purged from cache if unused by a User session Data Filter OR if not in the (history to look ahead production date range), when a production day is removed from cache the Schedules Grid view column "Cached" will change to No.

300000

Numeric

No

CacheBusinessActivityGroomInterval

Interval in milliseconds to groom the state and cumulative state counts for Business Activity.

12000

Numeric

No

CalcStatusCountsInterval

Interval in milliseconds to groom the status counts for Job Activity.

10000

Numeric

No

Logs generated during in-memory cache updates.

On successful initialization of the cache, these logs are seen in the ClientManager*.log:

06/04 07:31:53:593[1:main]: (mem=2030994776/2205155328) InMemoryDataCache: _storeGlobalValue CacheJACHistory: 5
06/04 07:31:53:593[1:main]: (mem=2030994776/2205155328) InMemoryDataCache: _
storeGlobalValue CacheJACLookAhead: 5
06/04 07:31:53:593[1:main]: (mem=2030994776/2205155328) InMemoryDataCache: _
storeGlobalValue CacheMsgLogNumRows: 10000
...........
06/04 07:31:54:207[1:main]: (mem=1913796472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache sysval Count : 37
06/04 07:31:54:241[1:main]: (mem=1849308472/2205155328) JobView: Resolving inherited fields for all 
JobView objects
06/04 07:31:54:247[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache job_view Count : 10
06/04 07:31:54:250[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in jobdependency_view Count : 0
06/04 07:31:54:260[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache owners_view Count : 7
06/04 07:31:54:263[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in varmst Count : 0
06/04 07:31:54:265[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in jobclass_view Count : 0
06/04 07:31:54:269[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache evntfscl Count : 1
06/04 07:31:54:277[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache secmst Count : 7
06/04 07:31:54:326[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache evntmst Count : 62
06/04 07:31:54:331[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache workgrp Count : 2
06/04 07:31:54:336[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in businessunit_view Count : 0
06/04 07:31:54:338[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in resmst Count : 0
06/04 07:31:54:351[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache servicemst Count : 24
06/04 07:31:54:361[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache trgmst Count : 1
06/04 07:31:54:374[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache usrmst Count : 5
06/04 07:31:54:378[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in actions_view Count : 0
06/04 07:31:54:381[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in imagemst Count : 0
06/04 07:31:54:385[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache viewmst Count : 1
06/04 07:31:54:387[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in foldermst Count : 0
06/04 07:31:54:389[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in trgjobcls Count : 0
06/04 07:31:54:392[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache evntcond Count : 9
06/04 07:31:54:403[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache evntchld Count : 62
06/04 07:31:54:414[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache messagecatalog_view Count : 59
06/04 07:31:54:456[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache javatz Count : 628
06/04 07:31:54:460[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache eventjobjoin_view Count : 1
06/04 07:31:54:467[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache eventactionjoin_view Count : 1
06/04 07:31:54:469[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache No data found in trgsch Count : 0
06/04 07:31:54:473[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache nodout Count : 2
06/04 07:31:54:484[1:main]: (mem=1817064472/2205155328) InMemoryDataCache: 
loadInitialDataIntoCache secmstservice Count : 168

These logs are seen in the clientmgr*.out at startup:

[06/04 07:31:52:971]:ClientNode: Primary Server = [tcp://localhost:6215] 
[06/04 07:31:52:971]:ClientNode: Backup Server = [null]
[06/04 07:31:53:080]:Retrieved a LogFile called 'production' 
[06/04 07:31:53:086]:Retrieved a LogFile called 'production' 
[06/04 07:31:53:236]:Retrieved a LogFile called 'production' 
[06/04 07:31:53:593]:Retrieved a LogFile called 'RegularFile' 
[06/04 07:31:53:927]:Retrieved a LogFile called 'RegularFile' 
[06/04 07:31:55:861]:Retrieved a LogFile called 'RegularFile' 
[06/04 07:32:11:868]:Adapters initialized: 17 seconds.
[06/04 07:32:11:868]:Client initialized.

The logs are generated by the failsafe code to check for duplicates schedules and message logs reload:

In Memory Cache Duplicate schedule for Prod Date: 2018-10-10 00:00:00.0 loadDataForMessageLog MessageLogs Count: 10000